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ABSTRACT 

'V 

Given  a  rectangular  matrix  A[x)  that  depends  on  the  independent  variables  z,  many  con¬ 
strained  optimization  methods  involve  computations  with  Z(x),  a  matrix  whose  columns  form  a 
basis  for  the  null  space  of  A(x).  When  A  is  evaluated  at  a  given  point,  it  is  well  known  that  a 
suitable  Z  (satisfying  AZ  =  0)  can  be  obtained  from  standard  matrix  factorizations.  However, 
Coleman  and  Sorensen  have  recently  shown  that  standard  orthogonal  factorization  methods  may 
produce  orthogonal  bases  that  do  not  vary  continuously  with  z;  they  also  suggest  several  tech¬ 
niques  for  adapting  standard  factorization  schemes  so  as  to  ensure  continuity  of  Z  in  the  neigh¬ 
borhood  of  a  given  point. 

(n  this  note,  discuss  several  aspects  of  the  representation  of  a  basis  Tor  the  null  space. 
We  describe  how  an  explicit  matrix  Z  can  be  obtained  at  any  point  using  a  method  for  updating 
a  factorization  with  either  Householder  or  stabilized  elementary  transformations.  Under  a  mild 
non-singularity  assumption,  the  elements  of  Z  arc  continuous  functions  of  z.  ViC  also  show 
that  the  chosen  f^rin  of  Z  is  convenient  and  ellicicnt  when  implementing  certain  methods  for 
nonlincarly  constrained  optimization. 
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1.  Introduction 

Given  an  mXn  matrix  A  (m  <  n),  many  constrained  optimisation  methods  involve  computations 
with  a  matrix  Z  whose  columns  form  a  basis  For  the  null  space  of  A.  it  is  well  known  that  certain 
factorizations  of  A  provide  stable  and  cificicnl  means  for  computing  Z.  We  illustrate  this  process 
with  the  TQ  factorization,  which  is  defined  by 

AQ  =  (  0  T  ),  (I) 

where  Q  is  an  n  X  n  nonsingular  matrix,  and  T  is  an  m  X  m  “reverse”  triangular  matrix  such  that 
Tij  =  0  for  i+j  <  m.  (The  reverse-triangular  form  of  T  has  advantages  in  implementing  certain 
constrained  optimization  methods;  see  Gill  cl  a I.,  1082).  If  A  has  full  rank,  T  is  nonsingular; 
in  this  case  it  follows  from  (I)  that  the  first  n  —  m  columns  of  Q  can  be  taken  as  the  columns 
of  the  matrix  Z.  If  Q  is  orthogonal,  the  TQ  factorization  is  simply  a  permutation  of  the  Qfi 
factorization  of  AT  (or  the  LQ  factorization  of  A). 

In  many  applications,  the  elements  of  A  are  functions  of  an  independent  variable  x  (x  €  S") 
-  for  example,  A  often  represents  the  Jacobian  of  a  set  of  constraints.  In  such  a  context,  it  is 
desirable  that  the  basis  for  the  null  space  can  be  represented  by  a  continuous  matrix  function  Z(x). 
For  example,  a  continuous  representation  of  Z  is  advantageous  in  proving  local  convergence  of 
methods  for  nonlinear  constraints  that  maintain  a  quasi-Newton  approximation  to  the  projected 
Hessian  of  the  Lagrangian  function,  since  the  operation  of  projection  is  defined  by  Z. 

Coleman  and  Sorensen  (1982)  recently  posed  the  following  question:  Let  8  be  a  ball  around 
a  point  x  (x  G  5R").  Suppose  that  A(x)  is  a  nmXn  matrix  of  rank  m  whose  elements  vary 
continuously  with  x  in  8.  Is  it  possible  to  construct,  in  a  stable  and  e/ficient  manner,  a  matrix 
Z(x)  with  elements  that  vary  continuously  with  x  in  8  f  They  observe  that  a  “standard”  method 
of  computing  the  orthogonal  factorization  through  Householder  matrices  may  not  provide  the 
required  continuous  matrix  Z(x),  ami  then  propose  several  alternative  strategies  for  ensuring  a 
continuous  Z  in  a  neighborhood  or  x. 

This  note  summarizes  the  main  features  of  a  scheme  for  computing  Z  that  has  been  used  in 
software  lor  quadratic  programming  and  nonlinear  programming  ((fill  et  a/.,  1983a  and  1983b).  In 
Section  2,  we  describe  the  algorithm  Tor  computing  Z.  Section  3  shows  that  the  method  produces  a 
continuous  basis  for  the  null  space  under  the  conditions  given  by  Coleman  and  Sorensen.  Finally, 
in  Section  4  we  discuss  some  advantages  or  the  chosen  representation  for  Z  in  the  context  of 
algorithms  for  constrained  optimization. 

We  emphasize  that  the  procedures  given  in  this  note  do  not  depend  on  use  of  the  TQ  rather 
than  other  factorizations.  We  have  chosen  to  specify  the  computations  in  terms  of  the  TQ 
factorization  in  order  to  describe  precisely  the  implementations  mentioned  above.  Use  of  the  TQ 
factorization  leads  to  no  increase  in  complexity  of  the  implementation,  although  it  does  slightly 
complicate  the  exposition. 


2.  Representation  and  computation  of  Z 

The  algorithm  for  computing  Z  is  essentially  a  method  for  updating  the  TQ  factorization  (I) 
when  the  matrix  Q  is  stored  explicitly  (see  Gill  et  a/.,  1982). 

We  assume  that  the  m  rows  or  A(x)  can  be  partitioned  into  two  groups:  the  first  mt  rows 
(denoted  by  AL)  arc  constant,  and  the  last  m*  rows  (denoted  by  /t*(*))  vary  with  x.  (This 
partition  corresponds  to  the  case  when  A[x)  is  the  Jacobian  of  a  mixture  of  linear  and  nonlinear 
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constraints.)  Thus,  A  and  T  in  (1)  have  the  forms 


where  Tl  and  TN  arc  reverse-triangular. 

Consider  now  a  different  matrix  A  that  is  not  necessarily  dose  to  A.  Observe  that  the  old  Q 
reduces  the  first  mL  rows  of  A  to  reverse- triangular  form,  i.e. 


AQ  = 


(£>-c  ;•> 


(2) 


where  S  is  mN  X  ( n-mL ).  In  order  to  obtain  the  new  TQ  factorization,  S  is  triangularized  by  a 
sequence  of  mN  transformations.  The  procedure  can  also  be  viewed  as  applying  mN  updates  to 
the  TQ  factorization  of  AL  while  the  rows  of  AN  are  added  one  at  a  time. 

The  triangularization  of  S  is  accomplished  in  a  standard  fashion  by  applying  a  sequence  of 
elementary  transformations  to  its  rows.  As  usual,  each  step  may  be  viewed  as  transforming  the 
“first”  row  of  a  successively  smaller  matrix.  To  illustrate  the  procedure,  consider  the  definition 
of  the  »-th  transformation  (»  =  l,...,mN).  At  this  stage,  rows  1  through  t  —  1  of  S  have  already 
been  triangularized,  and  the  corresponding  i  —  1  transformations  have  been  applied  to  the  rows 
of  S.  Let  j  denote  the  index  mL  +  i  —  1  (j  is  the  number  of  rows  of  A  already  in  reverse- triangular 
form).  Let  sf  denote  the  “reduced”  vector  to  be  triangularized  (i.e.,  the  first  n  —  j  components 
of  row  »),  and  assume  that  sj  is  given  by 

».r  =  (  ».T  «,  ),  (3) 


where  sj  is  a  vector  with  n  —  j  —  1  components  and  o<  is  the  t'-th  reverse-diagonal  element.  The 
triangularization  of  aj  is  achieved  by  an  elementary  matrix  J\  of  the  form 


Pi 


r  I  t 


w 


We  give  two  alternative  definitions  of  P,.  First,  P,  can  be  taken  as  a  Householder  matrix,  in 
which  case 

Ui  =  Vi  =  (oi  +  sign^.Mz  )  and  A  =  2,M*-  (5) 

(For  a  detailed  discussion  of  Householder  matrices,  sec  Stewart,  1973.)  If  regarded  as  a  function 
or  the  elements  of  the  matrix  P,  is  discontinuous  at  any  point  where  o,  vanishes. 

The  second  option  is  to  define  P,  as  an  elementary  but  non-orthogonal  matrix  (as  in  Gaussian 
elimination).  In  this  case, 


u,  =  en_y,  v,  =  (  **  ),  and  ft  =  eit  (6) 

where  eB_,  is  the  (n  —  j)-th  coordinate  vector.  In  constructing  a  stabilized  clemenLary  transfor¬ 
mation,  requirements  of  numerical  stability  forbid  the  use  of  a  pivot  element  <r,  that  is  “small 1 
relative  to  the  elements  of  lienee,  if  o,  is  loo  small,  a  column  interchange  must  be  performed, 
which  introduces  a  discontinuity  in  P<  as  a  function  of 


j 


3.  Continuity  of  Z 


Wo  now  place  Ihc  triangularixation  of  S  within  the  framework  of  obtaining  the  TQ  factorisa¬ 
tion  of  A.  The  successively  smaller  matrices  {Pi}  defined  above  arc  embedded  in  a  sequence  of 
n  X  n  matrices  {Pi},  i  =  defined  by 


(7) 


A  matrix  Qi  that  triangularizes  the  first  j  +  1  rows  of  A  is  given  by 

Qi  =  QPvPi  =  Qi-iPi, 


where  Q0  =  Q.  The  final  result  (after  m*  updates)  is 


AQmN 


(o  o  rL\ 

V  o  TN  w )' 


(8) 


so  that  Q  is  defined  by 

Q  =  =  QPi-PmH-  (») 

The  total  number  of  operations  required  to  obtain  T  and  Q  includes  the  following:  n2m„  to 
apply  the  transformations  to  S;  |mj,  +  tn*  (n  —  m)  to  perforin  the  reduction  with  Householder 
transformations  (}m’  +  ^m^(n  —  m)  with  elementary  matrices);  +2mN(n-m)  to  transform 
Q  with  Householder  matrices  (jm*  +mN(n  —  m)  with  elementary  matrices). 

S.  Continuity  of  Z 

In  this  section  we  sketch  a  proof  that  Z  as  computed  by  the  method  of  Section  2  is  continuous 
in  a  neighborhood  of  a  point  i  where  A{£)  has  full  rank. 

Let  An  denote  AN(£),  and  AN  denote  AN(x+6z),  where  z+6x  is  in  the  ball  B.  By  assumption, 
A  is  continuous,  and  hence 

An  =  A"  +  6A,  (10) 

where  6 A  can  be  made  as  small  as  desired  by  restricting  the  size  of  S. 

We  assume  that  the  TQ  factorization  (1)  of  A  is  given,  and  that  Q  is  bounded  (if  Q 
is  orthogonal,  the  latter  assumption  is  satisfied  automatically).  It  follows  from  (10)  and  the 
boundedness  of  Q  that  the  result  of  applying  Q  to  AN  is  a  matrix  that  is  “almost”  in  reverse- 
triangular  form.  Thus,  the  first  n  —  m,.  columns  of  ANQ  may  be  written  as 

7'  =  (0  T„)  +  6T,  (11) 

where  the  elements  of  ST  can  be  made  as  small  as  desired  by  appropriate  definition  of  3.  1/ct  t? 
denote  the  i-lh  row  of  T.  _Wc  now  consider  the  continuity  properties  of  the  matrices  {P,}  that 
will  be  applied  to  restore  T  to  reverse-triangular  form. 

The  continuity  argument  is  essentially  an  inductive  demonstration  that  the  “almost”  reverse- 
triangular  form  or  T  is  not  destroyed  by  application  of  the  transformations  {Pi}.  To  why,  consider 
the  special  character  of  the  Householder  transformation  that  reduces  the  first  row  of  T.  It  follows 
from  (II)  that,  when  B  is  sullicicnlly  small,  the  first  row  or  T  is  of  the  form 

<7  =  ( *T  *+<). 


(is) 
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where  6  is  n  vector  with  n  —  m,.  —  1  components,  t  is  the  first  reverse-diagonal  element  of  TN, 
and  ||£||  and  |c|  are  small.  From  (5),  the  Householder  vector  V|  is  given  by 

v  (  6 
vi  V  T  +  i  +  sign(T  +  f)||tl|| 

Since  A  has  full  rank,  r  is  non-zero,  and  hence  the  choice  of  sign  in  (13)  will  he  constant  in  a 
neighborhood  of  z.  Therefore,  tq  (and  Pi)  will  converge  to  a  limit  as  ||tfz||  approaches  zero. 

Now  consider  the  effect  of  Pi  on  the  remaining  rows  of  T.  It  follows  from  (4)  that  the  *-th 
row  transformed  by  l\  is  given  by 

where  n  =  i.T’vi/ft.  (14) 

Since  (13)  shows  that  the  first  n  —  mL  —  1  components  of  t>|  are  small,  (14)  indicates  that  the  effect 
of  transforming  T  by  P\  is  merely  to  add  small  perturbations  to  the  first  n  —  mL  —  1  components 
of  rows  2  through  mN.  Thus,  Tl\  remains  in  “almost”  reverse-triangular  form  and  a  relationship 
like  (12)  holds  for  the  next  row  to  be  triangularizcd.  Using  a  similar  argument,  we  can  show  that 
each  vector  i>,  is  continuous  in  a  neighborhood  of  z.  (In  the  limit,  (13)  shows  that  i \  becomes  a 
multiple  of  en_j.) 

An  analogous  argument  can  be  made  for  continuity  of  each  P,  defined  by  a  stabilized 
elementary  transformation.  In  this  ease,  it  can  be  shown  that  for  a  sufficiently  small  neighborhood, 
the  pivot  elements  do  not  become  too  small  because  they  arc  small  perturbations  of  the  reverse- 
diagonal  elements  of  T.  Using  (6)  and  (12),  we  also  see  that  the  vector  «,•  converges  to  zero  as 
||«z||  approaches  zero. 

Having  shown  that  the  matrices  {l\}  are  continuous  in  a  neighborhood  of  z,  we  must  now 
consider  the  continuity  of  Z.  Recall  from  (9)  that 

In  the  stabilized  elementary  ease,  it  is  easy  to  see  that  each  /\  converges  to  the  identity  matrix 
as  ||£z||  approaches  zero.  Hence,  Q  (and  Z)  are  continuous. 

Unfortunately,  the  same  simple  argument  does  not  apply  in  the  Householder  case,  since  it 
is  well  known  that  the  identity  matrix  is  not  a  Householder  matrix.  However,  because  the  first 
n  —  j  —  I  components  of  v*  arc  converging  to  zero,  the  application  of  Pt-  causes  changes  that 
converge  to  xcro  in  columns  1  through  n  —  j  —  1,  where  j  assumes  the  values  mt,...,m  —  1. 
Since  Z  comprises  the  first  n  —  m  columns  of  Q,  this  implies  that  the  changes  in  the  columns 
or  Z  converge  to  zero  as  ||<5z||  approaches  zero.  Thus,  Z  is  continuous  in  the  Householder  case, 
although  Q  is  not. 

Despite  the  continuity  of  Z,  it  may  be  considered  undesirable  for  the  remaining  columns  of  Q 
(and  the  reverse-diagonals  of  T)  to  undergo  the  changes  in  sign  that  result  when  a  Householder 
reduction  is  applied  to  a  nearly  triangular  matrix.  If  so,  the  difficulty  can  be  avoided  by 
systematically  changing  the  relevant  sign  after  each  transformation  is  applied  —  for  example, 
by  applying  the  composite  transformation  /«/-*»>  where  the  (n  —  j)-dimcnsional  diagonal  matrix 
D{  is  defined  by 

Di  —  diag(l, . . . ,  1,  —  1). 

With  this  technique,  T  and  Q  would  become  continuous  in  a  neighborhood  of  z.  Coleman  and 
Sorensen  (1982)  suggest  an  alternative  procedure  that  avoids  alternating  signs  in  the  diagonals  of 
T  and  the  last  m  columns  of  Q. 
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4.  Further  properties 

The  approach  of  Section  2  is  not  the  only  possible  way  to  obtain  a  representation  of  Z.  The 
most  obvious  alternative  is  to  apply  a  “standard”  Householder  triangularization.  In  this  case, 
the  Householder  vectors  are  stored  in  compact  form  along  with  T  in  an  n  X  m  matrix  (plus  an 
m-vcctor)  (sec,  c.g.,  Stewart,  1973),  so  that  no  explicit  matrix  Q  is  stored.  (Products  of  the  form 
Qv  or  QTv  are  obtained  by  applying  the  stored  transformations  to  v.) 

Assuming  that  the  rnL  transformations  corresponding  to  constant  rows  of  A  arc  retained,  the 
matrices  T  and  Q  of  Section  2  can  be  computed  using  the  standard  Householder  procedure  in 
mA,(2nmJ,  —  rnj)  operations  to  apply  the  mL  fi.<d  transformations  to  AN,  and  +m*  (»  — w) 
operations  to  produce  the  desired  reverse-triangular  form.  If  the  explicit  matrix  Q  is  needed,  it 
can  be  formed  (in  2nm(n  -  m)  +  |m3  operations)  by  multiplying  the  transformations  together  in 
reverse  order.  Thus,  the  standard  Householder  approach  requires  less  storage  and  work  than  the 
method  of  Section  2. 

Nonetheless,  an  explicit  representation  of  Q  has  certain  advantages  because  of  its  convenience 
and  llcxibility  under  more  general  circumstances  that  occur  in  constrained  optimization  methods. 
Our  particular  concern  is  to  perform  updates  efficiently  and  uniformly  when  simple  hounds  and 
linear  constraints  arc  treated  separately  from  nonlinear  constraints  —  as  in  the  methods  described 
in  Gill  ct  al.  (1983b).  Special  treatment  of  these  constraints  involves  not  only  exploiting  the 
presence  of  constant  rows  in  A  (which  can  be  done  with  the  standard  Householder  procedure,  as 
noted  above),  but  also  performing  operations  on  A  other  than  simply  adding  rows.  In  particular, 
a  given  matrix  A  will  often  be  subject  to  the  deletion  of  rows  and  the  addition  or  deletion  of 
columns  —  say,  in  computing  Lagrange  multiplier  estimates  for  different  versions  of  the  working 
set  at  the  same  point.  With  Q  stored  explicitly,  all  lour  types  or  updates  can  be  performed  in  an 
efficient  uniform  manner  (sec  Gill  cl  al.,  1982,  for  details).  However,  it  is  not  possible  to  perform 
the  latter  three  updates  within  the  framework  of  a  compact  representation  of  Q. 

Representing  each  change  to  Q  as  in  (8)  rather  than  recomputing  Q  from  a  compact  rep¬ 
resentation  is  useful  in  other  situations  -  for  example,  modifying  a  projected  Hessian  (or  ap¬ 
proximate  Hessian)  to  reflect  the  new  subspace  of  projection.  Finally,  an  explicit  representation 
of  Q  is  convenient  if  finite-differences  arc  to  be  taken  along  the  columns  of  Z. 
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Given  rectangular  matrix  A(x)  that  depends  on  the  Independent 
variables  x,  many  constrained  optimization  methods  involve  computations 
with  Z(x),  a  matrix  whose  columns  form  a  basis  for  the  null  space  of 
A(x).  When  A  is  evaluated  at  a  given  point,  it  is  well  known  that  a 
suitable  Z  (satisfying  AZ  -  0)  can  be  obtained  from  standard  matrix 
factorizations.  However,  Coleman  and  Sorensen  have  recently  shown  that 
standard  orthogonal  factorization  methods  may  produce  orthogonal  bases  that 
do  not  vary  continuously  with  x;  they  also  suggest  several  techniques  for 
adapting  standard  factorization  schemes  so  as  to  ensure  continuity  of  Z 
in  the  neighborhood  of  a  given  point. 

In  this  note,  we  discuss  several  aspects  of  the  representation  of  a 
basis  for  the  null  space.  We  describe  how  an  explicit  matrix  Z  can  be 
obtained  at  any  point  using  a  method  for  updating  a  factorization  with 
either  Householder  or  stabilized  elementary  transformations.  Under  a  mild 
non-singularity  assumption,  the  elements  of  Z  are  continuous  functions  of 
x.  We  also  show  that  the  chosen  form  of  Z  is  convenient  and  efficient 
when  Implementing  certain  methods  for  nonllnearly  constrained  optimization. 
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